Papers with Pile dataset
MiLe Loss: a New Loss for Mitigating the Bias of Learning Difficulties in Generative Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing generative language models neglect an inherent challenge in text corpus during training, i.e., the imbalance between frequent tokens and infrequent ones. |
| Approach: | They propose a function to mitigate the imbalance between frequent and infrequent tokens . authors propose 'MiLe Loss' function to assess learning difficulty of tokens during training . |
| Outcome: | Experiments show that models with proposed model can improve on downstream benchmarks. |
Impact of Pretraining Term Frequencies on Few-Shot Numerical Reasoning (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings. |
| Approach: | They investigate how well pretrained language models reason with terms less frequent in pretraining data. |
| Outcome: | The models are more accurate on instances whose terms are more prevalent, in some cases above 70% more accurate than the bottom 10%. |
Linear Transformers with Learnable Kernel Functions are Better In-Context Models (2024.acl-long)
Copied to clipboard
Yaroslav Aksenov, Nikita Balagansky, Sofia Lo Cicero Vaina, Boris Shaposhnikov, Alexey Gorbatovski, Daniil Gavrilov
| Challenge: | Current Language Models (LMs) lack essential In-Context Learning capabilities, a domain where the Transformer excels. |
| Approach: | They propose a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks. |
| Outcome: | The proposed model amplifies its In-Context Learning abilities on the Pile dataset. |
Flashback: Memory Mechanism for Enhancing Memory Efficiency and Speed in Deep Sequential Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing deep sequential processing models have problems with memory degradation and inaccurate gradient backpropagation. |
| Approach: | They propose a Flashback property that preserves memory as an identity mapping until it is overwritten by a hidden state at a different time step. |
| Outcome: | The proposed model can be implemented in Transformers and Mamba, and it performs well. |